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Abstract 

Purpose: To examine the construct validity and inter-rater reliability of the Neurological 
Impairment Scale (NIS) and compare ratings by medical and multidisciplinary teams in a mixed 
neurorehabilitation sample. To assess its concurrent and predictive validity as a predictor of 
outcome and functional gains during inpatient rehabilitation. Methods: The NIS was rated in a 
consecutive cohort of patients (n = 428) recruited from nine specialist neurorehabilitation units 
in London. Dimensionality and internal consistency were explored through principal compo- 
nents analysis with Varimax rotation. Inter-rater reliability and the relationship between NIS and 
functional outcome (UK Functional Assessment Measure (FIM + FAM)) were analysed in a sub- 
sample (n = 94) from one centre. Results: Factor analysis identified two principal domains 
("Physical" and "Cognitive") together accounting for 35% of the variance: their Cronbach's 
alpha values were 0.76 and 0.67, respectively. Inter-rater reliability was excellent for overall 
scores between doctors (ICC = 0.95 (95% CI = 0.91-0.97)) and acceptable between the medical 
and multidisciplinary team (ICC = 0.92 (95% CI = 0.88-0.95)). Change in NIS-physical score 
predicted 29% of the variance in functional gain (FIM + FAM change). Conclusion: These 
findings provide the first formal evidence for the validity and reliability of the NIS as a measure 
of neurological impairment for use in general neuro-rehabilitation settings. Its further 
application and exploration are now warranted. 
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>- Implications for Rehabilitation 

• The extent of neurological recovery occurring during rehabilitation can make an important 
contribution to functional gains. In order to interpret measurement of functional outcome, 
we need to be able to identify changes at the level of impairment. 

• Many of the available tools to measure severity of impairment are condition specific. The 
Neurological Impairment Scale (NIS) was developed for use across a broad range of disabling 
conditions alongside the UK FIM+FAM. 

• This first formal examination of its psychometric properties provides evidence for its 
scalability, reliability and validity. 

• The NIS has potential to provide useful information for case-mix adjustment and as a 
predictor of functional gain in general neurorehabilitation settings. 



Introduction 

The World Health Organisation's (WHO) International 
Classification of Functioning, Disability and Health (ICF), 
allows description of the impact of any health condition at a 
number of levels, including impairment of body structures and 
function, limitation in activities and restriction of participation 
[1]. Standardised measures of functional independence in daily 
living activities (such as the Barthel Index) [2], the Functional 
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Independence Measure [3-5] and Functional Assessment Measure 
(FIM + FAM) [6-8] are widely used across the Western world to 
evaluate global disability and the outcome of rehabilitation 
programmes. However, in neurological rehabilitation, patients 
present with diverse conditions and different combinations of 
physical, cognitive, communicative and behavioural impairments. 
They also present at varying stages of evolution of their condition, 
so that intervention may take place against a background 
trajectory of natural recovery (e.g. following acute stroke or 
brain injury) or deterioration (e.g. in progressive or neuro- 
degenerative disease). Functional gains are expected to be 
smaller in these latter conditions, but even the prevention of 
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deterioration can still improve the experience of patients and their 
families [9,10]. 

In order to make meaningful comparisons across different 
practices, programmes and populations, it is therefore helpful to 
have a standardised assessment of neurological impairment, against 
which any change in functional independence can be evaluated. A 
number of impairment sets exist for specific conditions, such as the 
NIH scale for stroke [11] and the ASIA scale for spinal cord injury 
[12]. However, these are not necessarily applicable across the 
broader range of neurological conditions. 

Within the UK, a national clinical dataset is now collated for 
all episodes of specialist in-patient rehabilitation through the UK 
Rehabilitation Outcomes Collaborative (UKROC) database [13]. 
The dataset gathers a common dataset of patient-level information 
on needs, inputs and outcomes across all conditions presenting for 
neurological rehabilitation. From April 2013, it is mandated for 
collection by all NHS specialist neurological rehabilitation 
services in England. The UK FIM + FAM is widely used in the 
UK as a global measure of disability [14] and has been adopted as 
the principal outcome measure for the UKROC dataset, and the 
Neurological Impairment Scale (NIS) as the measure of 
impairment. 

The NIS is not new. It evolved from the impairment set that was 
developed alongside the UK FIM + FAM in the 1990s, and has 
been included within the original "minimum dataset" of the UK 
FIM + FAM software programme since 1999 (see below for more 
details). Until now, however, there has been neither formal 
evaluation of its internal psychometric properties, nor of its 
usefulness as a predictor of disability or of potential to make func- 
tional gains in rehabilitation. Neither is it clear when and by whom 
the NIS should be recorded in the rehabilitation process. 

The aim of the present study was to perform a first 
examination of the reliability and validity of the NIS as a 
measure of an individual's specific and overall neurological 
impairment in a sample of patients with a diverse range of 
neurological disabilities. 

The specific objectives were: 

(a) to investigate its construct validity through an exploratory 
factor analysis and examination of internal consistency; 

(b) to determine inter-rater reliability and to compare ratings 
recorded by the medical staff and the multidisciplinary team; 

(c) to examine concurrent validity in terms of the relationship 
between changes in impairment and disability; 

(d) to assess predictive validity of the NIS as a predictor of 
outcome and functional gains made during inpatient 
rehabilitation. 

Methods 

The Neurological Impairment Scale 

The UK version of the FIM + FAM was developed in the mid- 
1990s, by the UK FIM + FAM Development Group, which 
consisted of experienced FIM + FAM users from 25 rehabilitation 
centres across the UK [8]. The group set out to improve 
consistency of scoring of the FIM + FAM between different 
centres, and to establish a central database to facilitate sharing of 
outcome data for neurological rehabilitation. As part of this 
development, the Group also identified a "minimum dataset" be 
collated alongside FIM + FAM data in a central database [15]. In 
keeping the WHO model of illness at the time - the International 
Classification of Impairment Disability and Handicap (ICIDH) 
[16] - the group developed a simple checklist of neurological 
impairments that were likely to be confounders of functional gain. 

This checklist provided a crude identification of different types 
of impairment, but it gave no indication of severity and was 
therefore insensitive to change even where neurological recovery 



(or indeed deterioration) was taking place. Consequently, later 
versions of the database not only expanded the range of 
impairments but also introduced a simple grading of severity 
("None", "Mild", "Moderate", "Severe"). After the 
International Classification of Functioning (ICF) was published 
in 2002 [1], the NIS items were mapped onto ICF codes to 
support data collection using the common language of the ICF. 

Over the course of some 15 years and several iterations, the 
tool has evolved into an ordinal measure of impairment that has 
potential applicability across a wide range of neurological 
conditions. The NIS (the NIS-version 8, as tested here), comprises 
17 items (each rated 0-2 or 0-3 giving a total score range 0-50). 
An example of the NIS score sheet and main scoring principles is 
given in Appendix 1. The full NIS together with detailed 
instructions for rating is available through the UKROC website 
(http://www.csi.kcl.ac.uk/ukroc). 

Design 

A cohort analysis of patients admitted for in-patient 
neurorehabilitation. 

In part 1, we used a principal components analysis with Varimax 
rotation to examine the dimensionality and internal consistency of 
the NIS in a multicentre sample (« = 428) from nine specialist 
neurorehabilitation units. 

In part 2, we performed a more detailed analysis of inter-rater 
reliability and the relationship between NIS and functional 
outcome (UK FIM + FAM scores) in the cohort sample (n = 94) 
from just one of the contributing centres. 

Ethics approval 

Approval for part 1 was granted by the Bromley Research Ethics 
Committee (Ref no: 09/H0805/25) and subsequently R&D centres 
of the seven participating NHS trusts. The remaining two 
recruiting centres were in the independent sector - approval for 
recruitment was obtained through their internal clinical and 
research governance processes. 

The centre in part 2 of the study gathers NIS and UK 
FIM + FAM data routinely in the course of clinical practice. 
Approval has been granted by the Harrow Research Ethics 
Committee (Ref no: 04/Q0405/81) for reporting these data 
retrospectively for research and audit purposes. 

Part 1: Construct validity (dimensionality and internal 
consistency) 

Participants 

For this part of the study, NIS ratings were examined at discharge 
from in-patient rehabilitation to maximise representation across 
the score range. Participants were recruited as part of a larger 
study of community rehabilitation services delivery post- 
discharge between September 2009 and March 2011 [17]. A 
consecutive cohort of 467 patients discharged over 1 year from 
each of nine tertiary neurorehabilitation units in London during 
this period were assessed for eligibility. Of these, 428 consented 
to participate and 403 had complete NIS scores, giving 25 with 
missing data. The demographic characteristics of the recruited 
sample (n = 428) are shown in Table 1. 

Data collection and analysis: 

NIS scores were recorded by the Multi-disciplinary (MD) treating 
team within 7 days before discharge to the community. Training in 
NIS rating was provided for teams who were not already familiar 
with the NIS as part of the UK FIM + FAM. 
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Table 1. Demographics of the study populations for Parts 1 and 2. 
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Part 1 


-n = 428 


Part 2 - 


« = 94 




Mean 


SD 


Mean 


SD 


Age (years) 


49.5 


15.3 


42.9 


14.5 


Length of stay (days) 






93 


62 


Mean time since onset (days) 


— 


— 


104 


71 


Males:Females 


n 

270:158 


% 

63:37 


n 

59:35 


% 

63:37 


Diagnosis 










Acquired brain injury 


315 


74% 


79 


84% 


Vascular (stroke, SAH) 


212 


50% 


41 


44% 


Traumatic 


63 


15% 


28 


30% 


Other (e.g. Hypoxic/inflammatory) 


40 


9% 


10 


11% 


Spinal cord injury 


38 


9% 


7 


l% 


Guillain-Barre and other peripheral neuropathies 


26 


6% 


5 




Multiple sclerosis 


21 


5% 


7 




Others 


27 


6% 


3 


3 '7, 



SAH = Sub-arachnoid Haemorrhage. 

Exploratory factor analysis was used to examine dimension- 
ality. The Kayser-Meyer-Olkin measure of sampling adequacy 
and Bartlett's test of sphericity both indicated that the correlation 
matrix was acceptable for factor analysis [18]. We used principal 
components analysis with Varimax rotation to identify the major 
sources of variance and to demonstrate whether a set of items can 
be meaningfully summarised as a single total score. Analysis was 
completed with IBM SPSS Statistics Package Version 19 
(Chicago, IL). After initial inspection of item total correlations 
and a scree plot, we extracted components with eigenvalues over 
2, which were then rotated using Varimax rotation. The internal 
consistency (i.e. Cronbach's a) was also calculated for the full 
scale and the first two principal components we identified. 

Part 2: Inter-rater reliability and the relationship between 
changes in impairment and disability 

Participants and data collection 

In this more detailed single-centre analysis we examined NIS and 
UK FIM + FAM scores on both on admission to, and discharge 
from, in-patient rehabilitation. This unit is the lead centre of 
training and dissemination of the UK FIM + FAM and NIS, so use 
of both scales is well established in this setting and all staff are 
fully trained in their use. The UK FIM + FAM is routinely 
recorded by the multidisciplinary team within 10 days of admis- 
sion and repeated within 7 days of discharge. The NIS is recorded 
by the MD Team for all patients during the admission 
FIM + FAM rating. During the study period it was also repeated 
at the discharge rating in n = 73 patients. 

In addition, the medical team independently undertook NIS 
ratings, based on presentation of findings collected at the admission 
medical clerking within 48 h of admission for « = 77 patients. To 
test inter-rater reliability of this method, two doctors independently 
recorded NIS scores for a sub-sample of n = 47 patients. 

Patients were included in the analysis sample if they had 

(a) paired NIS ratings on admission and discharge (« = 73) or 

(b) paired ratings by the medical and MD team (n = 77) 
(some patients were included on both counts). The demographic 
characteristics of this single-centre sample (total n = 94) are 
compared with that of the multicentre cohort in Table 1 . 

Analysis 

Where item scores were recorded as untestable (e.g. due to low 
awareness state, etc) scores of three were allocated to generate the 
total and subscale NIS scores. 



Reliability. Inter-rater agreement between doctors, and between 
the medical and MD teams, was evaluated using linear-weighted 
kappa coefficients computed in Stata version 10 (StataCorp, 
College Station, TX) and interpreted according to Landis and 
Koch [19]. Intra-class correlation (ICC) coefficients were used to 
compare total scores. 

All other statistical analysis was undertaken using IBM-SPSS 
v. 19. Because the NIS and UK FIM + FAM generate ordinal data, 
we report the median and interquartile range (25th-75th percent- 
ile) as well as means and standard deviations. However, as the 
samples were reasonably large and the data fell within acceptable 
limits of a normal distribution (one-sample Kolmogorov-Smirnov 
tests, /7>0.05), parametric statistics were applied throughout. 
Within-subjects effect sizes were calculated as Cohen's d with 
adjustment to allow for correlation between the means [20] using 
an online calculation package [21]. 

Concurrent validity. To examine the relationship between 
impairment and disability, we examined Pearson correlations 
between NIS and UK FIM + FAM scores on admission, and also 
between the change of scores from admission to discharge. 

It was anticipated that, in this general neuro-rehabilitation 
population, a significant proportion of patients would not 
demonstrate change in impairment. To determine whether func- 
tional change was greater in patients who changed at the level of 
impairment during rehabilitation, we divided the dataset retro- 
spectively into "impairment responders" (defined as those 
patients whose NIS scores reduced by 2 or more points between 
admission and discharge) and "impairment non-responders" 
(whose NIS score remained static or increased). Change in UK 
FIM + FAM scores was compared both within- (paired f-tests) 
and between (unrelated samples f-tests) these groups. 

Predictive validity. Stepwise multiple linear regression model- 
ling was used to evaluate MD team-rated NIS physical and 
cognitive subscale scores as predictors of functional status at 
discharge (UK FIM + FAM) and change in function during 
rehabilitation. Criteria for earning entry to the model were set at 
an alpha level of 0.05, and for removal at 0.1. 

Results 

Part 1: Exploratory factor analysis - dimensionality and 
internal consistency 

Table 2 presents the results of the principal components analysis 
after Varimax rotation. In fact the pattern of loadings on the first 
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two unrotated principal components and on the two rotated 
factors were almost identical. Two components had eigenvalues 
>2, which together accounted for 35% of the variance in total 
scores. The most significant source of variance is explained by a 
Physical component and the second major source of variance is 
a Cognitive component. Only two of the 15 items ("Hearing" 
and "Other") achieved loadings of <0.4 on both components 
mainly due to a preponderance of zero scores (90% and 85%, 
respectively). In subsequent analyses, "Hearing" was included 
in the cognitive subscale. The "Other" score (which most 
commonly include impairments such as seizures or pressure 
sores) was included in the physical subscale. Cronbach's alpha 
for the full 17-item NIS was 0.75. For the 10 Physical and seven 

Table 2. NIS score ranges, item-total correlations and loadings on first 
two principal components using Varimax rotation. 



Rotated component 
Descriptives matrix 



Median Item total Factor 1 Factor 2 

(IQR) Range correlations Physical Cognitive 



Eigen value 












3.6 


2.3 


% variance 












20.4 


14.6 


NIS Items 
















Left upper limb 


0 


(0-1) 


0- 


-3 


0.43** 


0.493 




Right upper limb 


1 


(0-2) 


0- 


-3 


0.44** 


0.502 




Left lower limb 


1 


(0-2) 


0- 


-3 


0.47** 


0.677 




Right lower limb 


1 


(0-2) 


0- 


-3 


0.50** 


0.693 




Trunk 


1 


(0-1) 


0- 


-2 


0.50** 


0.670 




Tone/contractures 


1 


(0-2) 


0- 


-3 


0.63** 


0.697 




Sensation 


1 


(0-2) 


0- 


-3 


0.55** 


0.629 




Perception 


0 


(0-1) 


0- 


-3 


0.38** 




0.498 


Speech and language 


1 


(0-2) 


0- 


-3 


0.45** 




0.580 


Cognitive 


1 


(1-2) 


0- 


-3 


0.33** 




0.808 


Behaviour 


0 


(0-0) 


0- 


-3 


0.23** 




0.635 


Mood 


1 


(0-1) 


0- 


-3 


0.40** 




0.526 


Vision 


1 


(0-1) 


0- 


-3 


0.31** 




0.562 


Hearing 


0 


(0-0) 


0- 


-2 


0.09 






Pain 


1 


(0-1) 


0- 


-3 


0.50** 


0.505 




Fatigue 


1 


(1-2) 


0- 


-3 


0.52** 


0.543 




Other 


0 


(0-0) 


0- 


-2 


0.12* 







** significant at p<0.001, * significant at p<0.05. 
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Cognitive items, Cronbach's alphas were 0.76 and 0.67, 
respectively. 

Part 2: Reliability and relationship with functional 
outcome 

(a) Inter-rater reliability 

Table 3 reports the level of inter-rater agreement found between 
the different ratings. Within the medical team there was high 
overall agreement between the two doctors reflected by a kappa 
coefficient of 0.81 for total NIS score and ICC 0.95 (95% 
confidence interval 0.91-0.97). Item-by-item agreement ranged 
from "substantial" to "almost perfect" with the exception of the 
fatigue item. 

Less strong agreement was expected between the ratings by the 
medical and MD team as up to 10 days elapsed between the 
assessments. Nevertheless, agreement in total scores was still 
acceptable (Kappa 0.69 for total NIS score, ICC 0.92 (95% CI 
0.88-0.95). Agreement for individual items was moderate to 
strong for 10/13 items. Only slight or fair agreement was 
observed, however, for the items for behavior and fatigue 
(see "Discussion" section). 

Figure 1 shows a box and whiskers plot of the total NIS scores 
rated on admission and discharge. Medical teams tended to record 
slightly lower ratings than the MD team, although this did not 
reach significance, either for the total scores or at item level. 
The MD team ratings were considered to be more reliable 
(see "Discussion" section) and were therefore used in the further 
evaluation of NIS as a predictor of functional outcome. 

(b) Concurrent validity — the relationship between impairment 
and disability 

Table 4 summarises the admission, discharge and change scores 
for the NIS and UK FIM + FAM scores, and Table 5 shows the 
correlation between them. Strong negative correlations were seen 
between NIS Physical and FIM + FAM Motor subscales on 
admission (Pearson r=— 0.86), and similarly between the 
respective cognitive subscales (r=— 0.90). Weaker, but still 
significant correlations were seen between physical and cognitive 
domains of the respective scales (r = 0.62, p < 0.001 in each case). 



Table 3. Inter-rater agreement: item-by-item linear- weighted kappa coefficients interpreted according to Landis and Koch [19]. 



Agreement between two doctors (n — 47) Agreement between Medical and MD Team scores (n = 94) 



Item 


Kappa 


95% CI* 


Interpretation 


Kappa 


95% CI 


Interpretation 


Left upper limb 


0.94 


0.72-1.0 


Almost perfect 


0.76 


0.61-0.92 


Substantial 


Right upper limb 


0.90 


0.69-1.0 


Almost perfect 


0.74 


0.59-0.90 


Substantial 


Left lower limb 


0.93 


0.72-1.0 


Almost perfect 


0.82 


0.66-0.97 


Almost perfect 


Right lower limb 


0.85 


0.65-1.0 


Almost perfect 


0.76 


0.92-0.59 


Substantial 


Trunk 


0.87 


0.64-1.0 


Almost perfect 


0.43 


0.27-0.59 


Moderate 


Tone 


0.63 


0.44-0.82 


Substantial 


0.45 


0.31-0.60 


Moderate 


Sensation 


0.82 


0.59-1.0 


Almost perfect 


0.65 


0.51-0.79 


Substantial 


Perception 


0.85 


0.60-1.0 


Almost perfect 


0.63 


0.47-0.79 


Substantial 


Speech 


0.90 


0.68-1.0 


Almost perfect 


0.81 


0.66-0.96 


Almost perfect 


Cognitive 


0.84 


0.63-1.0 


Almost perfect 


0.66 


0.52-0.79 


Substantial 


Behaviour 


0.94 


0.69-1.0 


Almost perfect 


0.13 


0-0.27 


Slight 


Mood 


0.75 


0.54-0.96 


Substantial 


0.57 


0.43-0.72 


Moderate 


Vision 


0.80 


0.56-1.0 


Substantial 


0.68 


0.52-0.84 


Substantial 


Hearing 


0.79 


0.52-1.0 


Substantial 


0.77 


0.59-0.95 


Substantial 


Pain 


0.76 


0.58-0.94 


Substantial 


0.43 


0.29-0.56 


Moderate 


Fatigue 


0.58 


0.40-0.77 


Moderate 


0.39 


0.27-0.52 


Fair 


Other 


0.84 


0.63-1.0 


Almost perfect 


0.44 


0.29-0.59 


Moderate 


Total NIS Kappa 


0.81 


0.63-0.99 


Almost perfect 


0.69 


0.56-0.95 


Substantial 


ICC 


0.95 


0.91-0.97 




0.92 


0.88-0.95 





*95% Confidence intervals (CI) were calculated as +/— 1.96 s standard error and the upper limit truncated at a maximum of 1.0. 
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Figure 1. Box and whisker plots for total NIS scores at admission and discharge. Figure 1 shows a box and whiskers plot of the total NIS scores as rated 
by the MD and medical teams on admission and the MD team at discharge. There was no significant difference between the two admission ratings 
(mean difference 0.03, 95% CI —1.16-1.11, f = 0.045, p = 0.96). However, there was a significant reduction in team-rated total scores between 
admission and discharge (see Table 4). 



Table 4. Descriptive statistics for NIS and UK FIM + FAM scores as rated by the MD Team on admission and discharge (n = 73). 



Paired sample t tests 

Admission Mean SD Median IQR Range t p ES* 



Neurological Impairment Scale (NIS) 



Physical Subscale 


14.0 


6.7 


13 


10- 


-19 


1-28 


Cognitive Subscale 


8.2 


6.2 


7 


4- 


-11 


0-21 


Total Score 


22.2 


11.6 


19 


15- 


-28 


4^18 



Discharge 



Physical Subscale 


11.8 


6.8 


11 


7-17 


0- 


-27 


Cognitive Subscale 


6.7 


5.6 


5 


2-8 


0- 


-21 


Total Score 


18.3 


11.5 


15 


10-22 


3-46 



Change 



Physical Subscale 


-2.2 


2.9 


-2 


-4-0 


-11-5 


6.4 


<0.001 


0.73 


Cognitive Subscale 


-1.5 


2.4 


-1 


-3-0 


-11-3 


5.3 


<0.001 


0.64 


Total Score 


-3.7 


4.1 


-3 


-7—1 


-18-5 


7.8 


<0.001 


0.90 



UK Functional Assessment Measure (FIM + FAM) 



Motor Subscale 


53.9 


30.0 


51 


21-86 


16- 


108 


Cognitive Subscale 


56.3 


27.0 


61 


32-79 


14 


-95 


Total Score 


110.1 


52.0 


123 


63-156 


30- 


189 



Discharge 



Motor Subscale 


74.3 


33.3 


87 


47-102 


16-111 


Cognitive Subscale 


69.5 


24.6 


77 


59-88 


14-98 


Total Score 


143.8 


55.6 


166 


109-188 


30-206 



Change 



Motor Subscale 


20.4 


16.5 


17 


6- 


-33 


-2- 


-62 


-10.5 


<0.001 


-1.26 


Cognitive Subscale 


13.3 


11.9 


11 


5- 


-18 


-5- 


-55 


-9.5 


<0.001 


-1.14 


Total Score 


33.7 


23.4 


31 


14-52 


-3- 


-93 


-12.3 


<0.001 


-1.48 



*Effect size (ES) calculated as Cohen's d allowing for the correlation between the mean. 
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Both the NIS and the FIM + FAM showed significant changes 
between admission and discharge for total scores and subscales 
(see Table 4). Changes in NIS scores were significantly correlated 
with change in their respective components of the UK 
FIM + FAM (r=— 0.51 to —0.56). However, although there was 
a strong negative correlation between change in the FIM + FAM 
Motor score and the NIS cognitive score on admission (r= —0.45, 
p< 0.001), no such relationship was seen with the admission NIS 
physical score. 

Within this dataset, approximately two-thirds (n = 49) were 
"impairment responders" (i.e. their NIS scores reduced by 2 or 
more points between admission and discharge. The remaining 24 
impairment ' 'non-responders' ' showed no such reduction - indeed 
10 of them showed an increase in total NIS score ranging from 1 
to 5. Both groups made significant functional gains during in- 
patient rehabilitation as illustrated in Figure 2. However, the 
' 'impairment ' 'responders' showed a significantly greater change 
in FIM + FAM motor score (mean change 24.8, sd 17.0) 
compared with "non-responders" (mean change 11.4, sd 11.0) 
giving a mean difference of 13.3 (95% CI 6.6, 20.0) 
t —4.4p< 0.001. Similarly, impairment responders showed a 
greater change in FIM + FAM cognitive score (mean change 16.6, 
sd 12.6) than the non-responders (mean change 6.5, sd 6.6) giving 
a mean difference of 10.0 (95% CI 5.5, 14.5) t -4.5/?<0.001. 

(c) Predictive validity — the NIS as a predictor of outcome and 
functional gains made during inpatient rehabilitation 

To determine whether the NIS adds to the prediction of functional 
outcome, two linear multiple regression models were tested using 
total discharge FIM + FAM score as the independent variable. 

In model 1, the FIM + FAM total admission score, and NIS 
physical and cognitive scores on admission all met the criteria 
for entry and were entered stepwise into the prediction model. 
The admission FIM + FAM score was entered first and accounted 
for 82% of the variance. The NIS cognitive score contributed 
a further 4% and the NIS physical score contributed 2%, so 
that together they predicted 88% of the total FIM + FAM 
discharge score. 

In model 2, in addition to the FIM + FAM total admission 
score, NIS physical and cognitive change scores from admission 
to discharge were entered stepwise into the model. Once again the 
admission FIM + FAM score accounted for 82% of the variance. 
The NIS physical change score accounted for a further 5% but the 
NIS cognitive change score was excluded as a predictor variable. 

To examine the factors predictive of change in function during 
rehabilitation, the following variables were entered stepwise into 
the model using the FIM + FAM total change score as the 
independent variable: (a) Admission FIM + FAM total score, 
(b) Admission NIS Physical and Motor scores, (c) NIS Physical 
and Motor change scores. Within this model, admission 
FIM + FAM score and admission NIS physical and cognitive 
scores and change in NIS cognitive score were all excluded 
as predictor variables. Only the change in Physical NIS score 
was entered into the model where it accounted for 29% of the 
variance. 

Discussion 

The NIS was developed as a measure of severity of neurological 
impairment across a broad range of disabling conditions. This 
first examination of its psychometric properties has provided 
evidence for its scaling properties, reliability and concurrent and 
predictive validity. Exploratory factor analysis in a large multi- 
centre sample demonstrated two distinct principal components, 
which led to the identification a 10-item sub-scale of physical 
impairment, and a 7-item sub-scale of cognitive impairment 
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_ FIM+FAM Motor 
0FIM+FAM Cognitive 



Non-responders n-24 Responders n=49 

NIS responder 

Figure 2. Box and whisker plots for change in total FIM + FAM domain scores between admission and discharge in the impairment "responder" and 
"non-responder" groups. 

Figure 2 shows box and whiskers plots of the FIM + FAM change scores, in patients who did and did not demonstrate change in the NIS score during 
their rehabilitation programme. Both groups improved overall, but impairment "responders" made significantly greater gains in both motor and 
cognitive function than the "non-responders". 



each with acceptable internal consistency. The "Hearing" and 
"Other" items did not load on either factor, reflecting the 
preponderance of zero scores in this sample. Whilst this might 
suggest a degree of item redundancy, when they do occur the 
impairments captured by these items (deafness, seizures, pressure 
sores) can potentially have substantial impact on rehabilitation. 
Therefore, they have been retained in the scale for their clinical 
importance. 

Inter-rater agreement between two doctors was very high for 
both the total score and in item-by-item analysis, confirming that 
the NIS is highly reliable when applied to the same information. It 
was expected that agreement between the medical and MD team 
ratings would be less strong, due to the time lag between the two 
assessments. Although overall agreement was still good and 
ratings were highly comparable for most items, agreement for 
some was marginal - in particular for behaviour and fatigue - and 
medical ratings tended to be lower. The MD team ratings were 
considered to be more reliable as they were based on a longer 
period of examination by a wider spectrum of clinicians. The 
findings suggest that, for clinical purposes, the NIS can be applied 
either by the medical team on admission or subsequently by the 
MD Team. However, the latter may provide a more comprehen- 
sive assessment and is perhaps more likely to identify subtle 
impairments, such as mood, pain, behaviour and fatigue as they 
emerge over time, with familiarity. 

The expected concurrent and divergent relationships were seen 
between the physical and cognitive domains of the NIS and 
UKFIM + FAM. However, the moderate correlations between 
these measures confirms that the underlying constructs of 
impairment and disability are distinct, each requiring measure- 
ment in their own right. 



As in other series that have used the FIM alone, the admission 
FIM + FAM score was the strongest predictor of outcome [22,23], 
and other authors have also noted the influence of baseline 
cognitive function on outcome [22,24]. In this series, functional 
gains during rehabilitation (as measured by change in total 
FIM + FAM) were significantly predicted by both the level of 
cognitive impairment on admission and by change in physical 
impairment (as measured by the NIS). To our knowledge, this is 
the first study to examine the influence of changing neurological 
impairment on functional gains recorded by UK FIM + FAM. Our 
findings suggest that the NIS can make a useful contribution to 
the prediction of functional outcome in a mixed-diagnosis group 
of patients with severe/complex neurological disability. 

The authors recognise a number of limitations to this study: 

(1) The findings from our exploratory factor analysis need to be 
confirmed in a wider population, preferably from centres 
outside London. 

(2) Further examination of the scaling properties of the NIS 
using modern psychometric techniques (such as the Rasch 
analysis) is required and indeed is underway. 

(3) Part 2 of this study was conducted in a single centre and 
needs to be expanded in a multi-centre analysis, across a 
range of clinical settings and with other samples of neuro- 
rehabilitation patients and clinicians. 

(4) The fact that a tool has been in use for some time does not 
necessarily mean that it is as good as it can be. As a result of 
this study, clinical teams have highlighted a number of areas 
of shortfall in the NIS including the evaluation of muscu- 
loskeletal impairment and bladder and bowel dysfunction. 
Further development is currently being explored to extend its 
scope and range. 



30 L. Turner-Stokes et al. 



Disabil Rehabil, 2014; 36(1): 23-31 



Nevertheless, the findings presented in this first psychometric 
analysis of the MS demonstrate it to be a promising measure of 
neurological impairment, suitable for use across a broad range of 
neurological conditions. They demonstrate that, even in its current 
form, the NIS can provide useful information for case-mix 
adjustment, over and above the admission FIM + FAM score, as a 
predictor of functional gain. Such information will assist the 
interpretation of functional outcomes from inpatient rehabilitation 
of people with complex neurological disabilities. 
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Appendix 1. Neurological Impairment Set for LTNC and Functional Categories. Version 8 



Name: 


Severity scores - extent to which deficit 


mpacts on function/engagement in rehab 






Score 


Impairment 


Functional impact 


No: 


n 
u 


None 


Normal function 




A 

i 


MEM 
Ivlllu 


Affecting high level function only 


Date of completion: 


2 


Moderate 


Significant limitation, but some useful functionl 


/ / 


J 


Severe 


Little or no useful function, effectively limiting rehabilitation 




(Circle one) 




Impairment Type 


Other type: 


O Motor 






O R Hemiparesis b7302 


Other: 


Left upper limb s730 


Subscore 




O L Hemiparesis b7302 




Right upper limb s730 


Subscore 




O Tetraparesis b7304 




Left lower limb s750 


Subscore 




O Paraparesis b7303 




Right lower limb s750 


Subscore 




O Monoparesis b7301 




Trunk s760 


Subscore 




O Ataxia b760 




Motor total score 


/14 








O Tone / joint range 


0 12 3 


(Untestable) 


O Spasticity b735 


Other: 








O Contractures b710 





O Sensation 


0 12 3 


(Untestable) O Somatic (eg touch) b265 


Other: 






O Propioception b260 








O Dysaesthesia b279 





O Perceptual function 


0 12 3 


(Untestable) O Neglect of body - b180 


Other: 






O Neglect of external space -b156 





O Speech and language 


0 12 3 


(Untestable) O Expressive b1 671 


Other: 






O Receptive b1670 








O Dysarthria b320 








O Cognitive speech b1670 





O Cognitive function 


0 12 3 


O Consciousness b110 


Other: 






O Orientation b1 14 








O Memory b144 








O Attention b140 








O Initiation b147 








O Executive function b164 








eg insight, planning, flexible thought 





O Behaviour 


0 12 3 


(Untestable) O Verbal aggression d7202 


Other: 






O Physical aggression d7202 








O Disinhibition d7202 





O Mood 


0 12 3 


(Untestable) O Depression / Low mood b152 


Other: 






O Anxiety b152 








O Emotional lability b1521 





O Seeing and Vision 


0 12 3 


(Untestable) O Visual field loss b21 01 


Other: 






O Uncorrectable acuity b2100 








O Double vision b2152 





O Hearing 


0 12 3 


O Sensorineural (ICF- no code identified) 


Other: 






O Conductive (ICF- no code identified) 





O Pain 


0 12 3 


(Untestable) 


O Neuropathic pain b280 

O Musculoskeletal pain b280 

O Pain due to spasticity b280/b735 


Other: 




O Fatigue 


0 12 3 


(Untestable) 


O Reduced cardiovascular fitness 

O Muscle fatiguabiity b740 

O Cognitive fatigue (ICF- no code identified) 


Other: 



O Other 


0 12 3 


O Seizures (ICF- no code identified ) 


Other: 






O Pressure sores b820 





TOTAL SCORE 




Range 0-50 





